[Bugfix][Router] Run KvawareRouter /tokenize fallback in executor to avoid blocking event loop - #1069
Conversation
…ng event loop The remote /tokenize fallback in KvawareRouter.route_request() used a synchronous requests.post call directly on the event loop. Under load, this blocks the loop and can stall health checks and other requests. Mirror the tokenize_prompt() fallback pattern by running the blocking HTTP call via run_in_executor. Signed-off-by: Asthenia <asthenia0412@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request wraps a blocking HTTP POST request in an executor to prevent stalling the event loop in route_request. The reviewer suggests leveraging the shared aiohttp client session instead of running requests.post in an executor, which avoids thread-pool overhead and prevents socket exhaustion through connection reuse.
| # Run the blocking HTTP call in an executor so it does not | ||
| # stall the event loop (mirrors tokenize_prompt fallback). | ||
| loop = asyncio.get_running_loop() | ||
| response = await loop.run_in_executor( | ||
| None, | ||
| lambda: requests.post( | ||
| remote_url, headers=headers, json=data, timeout=10 | ||
| ), | ||
| ) | ||
| token_ids = response.json()["tokens"] |
There was a problem hiding this comment.
Since route_request is an asynchronous function and has access to the request object, we can leverage the shared aiohttp client session from request.app.state.aiohttp_client_wrapper() instead of using requests.post in an executor.
Using requests.post without a session creates a new TCP connection for every fallback request, which is highly inefficient under load and can lead to socket exhaustion. Utilizing the shared aiohttp client session enables connection reuse and avoids the overhead of thread-pool execution.
client = request.app.state.aiohttp_client_wrapper()
async with client.post(
remote_url, headers=headers, json=data, timeout=10
) as response:
response_json = await response.json()
token_ids = response_json["tokens"]…ze fallback Per review feedback, replace the executor-wrapped requests.post with the shared aiohttp client session from request.app.state.aiohttp_client_wrapper(). This reuses TCP connections (avoiding socket exhaustion under load) and keeps the call off the event loop. Signed-off-by: Asthenia <asthenia0412@gmail.com>
|
Thanks for the review! I updated the fallback to use the shared |
1 participant
Description
KvawareRouter.route_request()uses a synchronousrequests.postcall for the remote/tokenizefallback directly on the event loop. Under load, this blocking call stalls the event loop and can delay health checks and other concurrent requests.The same pattern was already fixed in
tokenize_prompt()by running the blocking HTTP call viaasyncio.get_running_loop().run_in_executor(). This PR applies the same fix to theroute_request()fallback for consistency.Change
src/vllm_router/routers/routing_logic.pyTesting
ast.parse